Document watcher mode dbt 1.5+ requirement by tatiana · Pull Request #2700 · astronomer/astronomer-cosmos

tatiana · 2026-05-19T10:43:03Z

Summary

Add a "Minimum supported dbt version" subsection to docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst documenting that ExecutionMode.WATCHER requires dbt-core 1.5+.

Why dbt 1.5+

ExecutionMode.WATCHER passes --log-format json to dbt after the subcommand (e.g. dbt build --log-format json …) so the producer and the consumer-fallback path can consume structured per-model events. dbt-core's Click-based CLI, introduced in 1.5, declares --log-format on the parent group and re-exposes it on every subcommand via a shared global_flags decorator, so dbt run --log-format json parses successfully from 1.5 onward. See the dbt-core 1.5.0 changelog ("Implemented new log cli parameters for finer-grained control", tracked in dbt-core#6639; full entry in CHANGELOG.md).

dbt versions older than 1.5 are EOL per the dbt Core versions page; the doc note recommends ExecutionMode.LOCAL for users who must run an older dbt.

Note on PR history: An earlier revision of this PR claimed the floor was dbt 1.8+, on the theory that --log-format is global-only on 1.5/1.6/1.7. That was empirically wrong — dbt run --log-format json parses successfully on dbt-core 1.5 in a fresh venv. Three reference-only requirements-airflow-3.2-dbt-1.{8,9,10}.txt snapshots originally committed to support the 1.8+ claim have been dropped from the PR; only the doc addition remains. The companion PR #2704 (which translated the assumed No such option '--log-format' error into a "use dbt 1.8+" hint) has also been closed since the underlying error doesn't actually occur on any dbt-core release Cosmos supports.

Test plan

hatch run docs:build succeeds with no new warnings (the two docs/guides/run_dbt/customization/scheduling.rst duplicate-target warnings predate this branch).
dbt run --log-format json empirically verified to parse on a dbt-core 1.5 venv — confirms the floor stated in the doc.

🤖 Generated with Claude Code

``ExecutionMode.WATCHER`` works only on dbt-core 1.8 or newer. On older dbt versions the producer and consumer-fallback paths fail at the dbt CLI layer with ``No such option '--log-format'`` before any of the watcher event-handling code runs. dbt < 1.8 has also reached end of life and is no longer supported by dbt Labs. The watcher uses ``--log-format json`` so the producer and the consumer fallback receive structured per-model events from dbt. Cosmos passes the flag after the dbt subcommand (e.g. ``dbt build --log-format json …``). On dbt 1.5 / 1.6 / 1.7 ``--log-format`` is a global flag only -- it must appear before the subcommand on those versions, so the dbt CLI parser rejects the command. dbt 1.8 was the release that added ``--log-format`` as a per-subcommand flag, making the current command shape work. This limitation is unrelated to and predates #2698 (the BOSS-401 retry fix); it surfaced while running the regression test on the full dbt matrix and is worth surfacing in the watcher docs so users on older dbt aren't confused by the ``No such option`` error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

While documenting the minimum-dbt-version constraint for watcher mode (``ExecutionMode.WATCHER`` requires dbt-core 1.8+), the watcher regression test from #2684 was rerun against AF 3.2 with each supported dbt version and the resulting environment captured via ``pip freeze``. The four snapshots in ``requirements/`` (1.8.9, 1.9.10, 1.10.21, plus the existing 1.11.7) are reference-only: ``pre-install-airflow.sh`` still pins ``requirements-airflow-3.2-dbt-1.11.txt`` regardless of the matrix ``dbt`` value, so creating an env like ``tests.py3.10-3.2-1.8`` will still install dbt 1.11. The new files document a known-good dependency set that reviewers and CI can use to reproduce the manual validation; wiring them into the matrix would be a separate change. Each snapshot was produced after overriding dbt-core in a freshly created hatch env: :: hatch run tests.py3.10-3.2-1.8:pip install --force-reinstall \\ 'dbt-core~=1.8.0' 'dbt-postgres~=1.8.0' 'dbt-duckdb~=1.8.0' hatch run tests.py3.10-3.2-1.8:pip freeze \\ > requirements/requirements-airflow-3.2-dbt-1.8.txt then verifying the watcher regression test passes: :: AIRFLOW_HOME=$(pwd)/dev \\ hatch run tests.py3.10-3.2-1.8:test-integration \\ --ignore=tests/test_example_dags.py \\ --ignore=tests/test_example_dags_no_connections.py \\ -k test_dbt_task_group_watcher_retry_recovers_skipped_downstream dbt 1.5 / 1.6 / 1.7 are not included: the watcher's ``dbt build --log-format json …`` command shape is rejected by their CLI parsers, so no valid requirements set produces a passing test on those versions (see the docs note in this same change). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Documents the pre-existing limitation that ExecutionMode.WATCHER requires dbt-core ≥ 1.8 (due to --log-format json flag parsing behavior in older dbt), and adds dependency “snapshot” files intended to capture the exact Airflow 3.2 + dbt 1.8/1.9/1.10 environments used for watcher regression validation.

Changes:

Add a “Minimum supported dbt version” subsection to the watcher execution mode docs clarifying dbt-core 1.8+ requirement and the --log-format CLI behavior difference.
Add reference dependency snapshots for Airflow 3.2 with dbt 1.8, 1.9, and 1.10 under requirements/.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst	Adds documentation clarifying dbt-core ≥ 1.8 requirement for `ExecutionMode.WATCHER`.
requirements/requirements-airflow-3.2-dbt-1.8.txt	Adds an Airflow 3.2 + dbt 1.8 dependency snapshot.
requirements/requirements-airflow-3.2-dbt-1.9.txt	Adds an Airflow 3.2 + dbt 1.9 dependency snapshot.
requirements/requirements-airflow-3.2-dbt-1.10.txt	Adds an Airflow 3.2 + dbt 1.10 dependency snapshot.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

The pip-freeze snapshots emitted an ``-e git+ssh://…@<sha>#egg=astronomer_cosmos`` line because the hatch envs installed cosmos in editable mode against the local working tree. Drop those lines so the new files match the shape of the existing ``requirements-airflow-3.2-dbt-1.11.txt``, which omits the cosmos pin -- the consuming reader is expected to install the package they're testing separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

codecov · 2026-05-19T11:11:51Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.04%. Comparing base (d6956b7) to head (aab5c4f).
⚠️ Report is 20 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2700   +/-   ##
=======================================
  Coverage   98.04%   98.04%           
=======================================
  Files         105      105           
  Lines        7864     7864           
=======================================
  Hits         7710     7710           
  Misses        154      154

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pankajastro · 2026-05-19T12:10:46Z

But log-format flag was added very early PR: dbt-labs/dbt-core#1791, no?

pankajkoti

I see we created this PR to address PR comment https://github.com/astronomer/astronomer-cosmos/pull/2684/changes/BASE..79c973fe977504cfc2249cf70d85e77be47b4946#r3248651514

I guess my comment there was unclear. I just wrote another response there to clarify in https://github.com/astronomer/astronomer-cosmos/pull/2684/changes/BASE..79c973fe977504cfc2249cf70d85e77be47b4946#r3266181652.

Also, are we intending to use the requirements files added here in some tests, or does our CI already run them? If not, can we add a note somewhere on the purpose of checking in these files?

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

The previous explanation under "Minimum supported dbt version" claimed dbt 1.5 / 1.6 / 1.7 reject ``--log-format`` after the subcommand and that the per-subcommand flag was only added in 1.8. Empirically that's wrong: dbt's Click-based CLI, introduced in 1.5, declares ``--log-format`` on the parent group and re-exposes it on each subcommand via a shared global-flags decorator, so ``dbt run --log-format json`` parses successfully from 1.5 onward. Rewrite the paragraph to state the actual floor (dbt 1.5, when the Click CLI landed) and link the dbt-core 1.5.0 changelog entry that introduced the finer-grained log CLI parameters (dbt-core#6639). Keep the EOL reference and the ``ExecutionMode.LOCAL`` fallback for users on older dbt. 🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

tatiana · 2026-05-27T10:45:53Z

Thanks a lot for the feedback, @pankajastro and @pankajkoti .

I had the impression in that PR that they had set --log-format as a global flag rather than a subcommand flag, but I read the code and realised that the subcommands inherited this flag as well. So I changed the version from 1.8 to 1.5. I validated that it works with 1.5. I closed the other PR I had created regarding the messaging in Cosmos, since this is an old version of dbt that reached EOL.

Finally, I removed the requirements.txt I used to validate the behaviour. Thanks, @pankajkoti , for clarifying the message on the other PR.

The three reference-only snapshot files were added to support a "watcher requires dbt 1.8+" claim that no longer holds: dbt-core's Click-based CLI accepts ``--log-format json`` after the subcommand from 1.5 onward (see dbt-core 1.5.0 changelog, dbt-core#6639), so the watcher's effective floor is 1.5+ and there's no behavioural reason to fix the snapshots at 1.8 / 1.9 / 1.10 in this branch. Keep ``requirements-airflow-3.2-dbt-1.11.txt`` — it predates this PR and is referenced by ``scripts/test/pre-install-airflow.sh``. 🤖 Generated with Claude Code (https://claude.com/claude-code)

pankajastro

LGTM.

But as per our policy, the minimum supported dbt-core version is 1.5. Wondering if highlighting this to the watcher would be redundant? Approving — please feel free to decide whether this is really needed or not.

tatiana and others added 2 commits May 19, 2026 11:29

Copilot AI review requested due to automatic review settings May 19, 2026 10:43

tatiana requested review from a team, corsettigyg, dwreeves and jbandoro as code owners May 19, 2026 10:43

tatiana requested review from pankajastro and pankajkoti May 19, 2026 10:43

tatiana temporarily deployed to internal May 19, 2026 10:43 — with GitHub Actions Inactive

Copilot started reviewing on behalf of tatiana May 19, 2026 10:43 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

Comment thread requirements/requirements-airflow-3.2-dbt-1.8.txt Outdated

Comment thread requirements/requirements-airflow-3.2-dbt-1.9.txt Outdated

Comment thread requirements/requirements-airflow-3.2-dbt-1.10.txt Outdated

tatiana temporarily deployed to internal May 19, 2026 10:52 — with GitHub Actions Inactive

This was referenced May 19, 2026

Consolidate watcher-related dbt projects under dev/dags/dbt/ #2701

Open

Watcher: Surface watcher dbt 1.8+ requirement in the error message #2704

Closed

pankajkoti reviewed May 19, 2026

View reviewed changes

Comment thread requirements/requirements-airflow-3.2-dbt-1.10.txt Outdated

Comment thread requirements/requirements-airflow-3.2-dbt-1.8.txt Outdated

Comment thread requirements/requirements-airflow-3.2-dbt-1.9.txt Outdated

tatiana commented May 27, 2026

View reviewed changes

Comment thread docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst Outdated

Apply suggestion from @tatiana

aab5c4f

Copilot AI review requested due to automatic review settings May 27, 2026 08:35

tatiana temporarily deployed to internal May 27, 2026 08:35 — with GitHub Actions Inactive

Copilot started reviewing on behalf of tatiana May 27, 2026 08:35 View session

tatiana commented May 27, 2026

View reviewed changes

Comment thread docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst Outdated

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst Outdated

tatiana temporarily deployed to internal May 27, 2026 10:27 — with GitHub Actions Inactive

tatiana assigned pankajastro May 27, 2026

tatiana changed the title ~~Document watcher mode dbt 1.8+ requirement~~ Document watcher mode dbt 1.5+ requirement May 27, 2026

Update requirements/requirements-airflow-3.2-dbt-1.8.txt

f80bccf

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Copilot AI review requested due to automatic review settings May 27, 2026 10:41

Update requirements/requirements-airflow-3.2-dbt-1.9.txt

f7cdbdc

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Copilot started reviewing on behalf of tatiana May 27, 2026 10:41 View session

Update requirements/requirements-airflow-3.2-dbt-1.10.txt

e79acc1

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Copilot AI reviewed May 27, 2026

View reviewed changes

tatiana temporarily deployed to internal May 27, 2026 10:46 — with GitHub Actions Inactive

tatiana temporarily deployed to internal May 27, 2026 10:47 — with GitHub Actions Inactive

pankajastro approved these changes May 27, 2026

View reviewed changes

tatiana merged commit 4ce3ada into main May 28, 2026
20 checks passed

tatiana deleted the docs-watcher-min-dbt-version branch May 28, 2026 09:29

tatiana added the roadmap:P1 BOSS roadmap-committed work (priority P1) label Jun 6, 2026

Conversation

tatiana commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why dbt 1.5+

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pankajastro commented May 19, 2026

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana commented May 27, 2026

Uh oh!

pankajastro left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tatiana commented May 19, 2026 •

edited

Loading

codecov Bot commented May 19, 2026 •

edited

Loading

pankajastro left a comment •

edited

Loading